Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(Don't merge until we find the right format for depth)This PR improves speed of simgetimages by :
I've made changes from RGBA->RGB in the client side code I could find.
An added benefit of RGBA -> RGB is that the image responses from the simgetimages() API match the unreal subwindow previews nicely. This has been a complaint by our users in the past, in that the simgetimage() responses tend to be much darker than the preview in the subwindow
Let's talk about RGB first:
Behind the scenes, Unreal initializes the render target to RGBA16f as can be seen in the engine code here
The suggestion in Improve image capture speed #1818 works well in that we can initialize it to
EPixelFormat::PF_B8G8R8A8
.There's also an
EPixelFormat::PF_R8G8B8A8_UINT
, but trying that makes the engine crash with assert failure in that the requested render target format is not supported.Interestingly,
PF_R8G8B8A8
leads to the same assert failure.There's no
PF_B8G8R8A8_UINT
, unfortunately.Side note: these names are misleading in that BGR is actually RGB - https://answers.unrealengine.com/questions/69615/are-epixelformats-pf-b8g8r8a8-and-pf-r8g8b8a8-flip.html
Why engine crashes though, is something I am not sure about.
Q. We should try to get int8 images / textures? We can hopefully change the scene capture to get rgba int (not float) and then use
PF_R8G8B8A8_UINT
? Although I am not sure if this will help coz the GPU would have the images as floats only.Moving on to depth:
Now, there should be a similar optimization possible for render targets for depth images. As far as I understand, it should be
EPixelFormat::PF_R16F
as alluded by this line https://github.com/Microsoft/AirSim/blob/master/Unreal/Plugins/AirSim/Source/RenderRequest.cpp#L127. However doing so, again leads to engine crashing with assert failure in that the requested render target format is not supported .Not completely sure why is that++
My hunch is this happens coz the capture source is being initialized to
ESceneCaptureSource::SCS_FinalColorLDR
for all image types at this line https://github.com/Microsoft/AirSim/blob/master/Unreal/Plugins/AirSim/Source/PIPCamera.cpp#L66.ESceneCaptureSource::SCS_FinalColorLDR
doesn't seem efficient for depth images. I tried changing it toESceneCaptureSource::SCS_SceneDepth
, similarly there's aESceneCaptureSource::SCS_Normal
which should be used for the normal image?Unfortunately, engine crashes yet again with those two options.
Anyhow, for now, I have added a placeholder TMap for image_types to pixel_format types. The map is only used for all non depth image types (aka
image_type==0 || image_type==5 || image_type==6 || image_type==7
).for depth images, I still call
initautoformat
for now.P.S. Technically, we should use
ETextureRenderTargetFormat
, which is the set of EPixelFormats exposed to rendertarget.However, the mapping is fairly intuitive from the variable names as can be seen in the engine code in these lines, so, i am sticking with pixelformat